Linear Program Approximations for Factored Continuous-State Markov Decision Processes
نویسندگان
چکیده
Approximate linear programming (ALP) has emerged recently as one of the most promising methods for solving complex factored MDPs with finite state spaces. In this work we show that ALP solutions are not limited only to MDPs with finite state spaces, but that they can also be applied successfully to factored continuous-state MDPs (CMDPs). We show how one can build an ALP-based approximation for such a model and contrast it to existing solution methods. We argue that this approach offers a robust alternative for solving high dimensional continuous-state space problems. The point is supported by experiments on three CMDP problems with 24-25 continuous state factors.
منابع مشابه
Overview of Linear Program Approximations for Factored Continuous and Hybrid-State Markov Decision Processes
Approximate linear programming (ALP) is as one of the most promising methods for solving complex factored MDPs. The method was applied first to tackle problems with discrete state variables. More recently the ALP methods that can solve MDPs with continuous and hybrid (both continuous and discrete) variables have emerged. This paper briefly reviews the work on ALP methods for such problems.
متن کاملOn the Smoothness of Linear Value Function Approximations
Markov decision processes (MDPs) with discrete and continuous state and action components can be solved efficiently by hybrid approximate linear programming (HALP). The main idea of the approach is to approximate the optimal value function by a set of basis functions and optimize their weights by linear programming. It is known that the solution to this convex optimization problem minimizes the...
متن کاملUsing mathematical programming to solve Factored Markov Decision Processes with Imprecise Probabilities
This paper investigates Factored Markov Decision Processes with Imprecise Probabilities (MDPIPs); that is, Factored Markov Decision Processes (MDPs) where transition probabilities are imprecisely specified. We derive efficient approximate solutions for Factored MDPIPs based on mathematical programming. To do this, we extend previous linear programming approaches for linear approximations in Fac...
متن کاملHeuristic Refinements of Approximate Linear Programming for Factored Continuous-State Markov Decision Processes
Approximate linear programming (ALP) offers a promising framework for solving large factored Markov decision processes (MDPs) with both discrete and continuous states. Successful application of the approach depends on the choice of an appropriate set of feature functions defining the value function, and efficient methods for generating constraints that determine the convex space of the solution...
متن کاملAn MCMC Approach to Solving Hybrid Factored MDPs
Hybrid approximate linear programming (HALP) has recently emerged as a promising framework for solving large factored Markov decision processes (MDPs) with discrete and continuous state and action variables. Our work addresses its major computational bottleneck – constraint satisfaction in large structured domains of discrete and continuous variables. We analyze this problem and propose a novel...
متن کامل